Evaluation of the CMU ATIS System

نویسنده

  • Wayne H. Ward
چکیده

The CMU Phoenix system is an experiment in understanding spontaneous speech. It has been implemented for the Air Travel Information Service task. In this task, casual users are asked to obtain information from a database of air travel information. Users are not given a vocabulary, grammar or set of sentences to read. They compose queries themselves in a spontaneous manner. This task presents speech recognizers with many new problems compared to the Resource Management task. Not only is the speech not fluent, but the vocabulary and grammar are open. Also, the task is not just to produce a transcription, but to produce an action, retrieve data from the database. Taking such actions requires parsing and "understanding" the utteraoce. Word error rate is not as important as utterance understanding rate. Phoenix attempts to deal with phenomena that occur in spontaneous speech. Unknown words, restarts, repeats, and poody formed or unusual grammar are common is spontaneous speech and are very disruptive to standard recognizers. These events lead to misrecognitions which often cause a total parse failure. Our strategy is to apply grammatical constraints at the phrase level and to use semantic rather than lexical grammars. Semantics provide more constraint than parts of speech and must ultimately be delt with in order to take actions. Applying constraints at the phrase level is more flexible than recognizing sentences as a whole while providing much more constraint than word-spotting, Restarts and repeats are most often between phase occurences, so individual phrases can still be recognized correctly. Poorly constructed grammar often consists of well-formed phrases, and is often semantically well-formed. It is only syntactically incorrect. We associate phrases by frame-based semantics. Phrases represent word strings that can fill slots in frames. The slots represent information which the frame is able to act on. The current Phoenix system uses a bigram language model with the Sphinx speech recognition system. The top-scoring word string is passed to a flexible frame-based parser, The parser assigns phrases (word strings) from the input to slots in frames. The slots represent information content needed for the frame. A beam of frame hypotheses is produced and the best scoring one is used to produce an SQL query. I N T R O D U C T I O N Understanding spontaneous speech presents several problems not found in transcribing read speech input. Spontaneous speech is often not fluent. It contains stutters, filled pauses, restarts, repeats, interjections, etc. Casual users do not know the lexicon and grammar used by the system. It is therefore very difficuk for a speech understanding system to achieve good coverage of the lexicon and grammar that subjects might use. Also, the task of the system is not just to produce a transcription, but to produce an action. Taking such actions requires parsing and "understanding" the utterance. Word error rate is not as important as utterance understanding rate. The Air Travel Information Service task is being used by several DARPA-funded sites to develop and evaluate speech understanding systems for database query tasks. In the ATIS task, novice users are asked to perform a task that requires getting information from the Air Travel database. This database contains information about flights and their fares, airports, aircraft, etc. The only input to the system is by voice. Users compose the questions themselves, and are allowed to phrase the queries any way they choose. No explicit grammar or lexicon is given to the subject. At Carnegie Mellon University, we have been developing a system, called Phoenix, to understand spontaneous speech [1] [2] [3]. We have implemented an initial version of this system for the ATIS task, This paper presents the design of the Phoenix system and its current status. We also report system evaluation results for the DARPA Feb91 test. T H E P H O E N I X S Y S T E M Some problems posed by spontaneous speech are: • User noise breath noise, filled pauses and other user generated noise • Environment noise door slams, phone rings, etc. • Out-of-vocabulary words The subject says words that the system doesn't know. • Grammatical coverage Subjects often use grammatically ill-formed utterances and restart and repeat phrases. Phoenix address these problems by using non-verbal sound models, an out-of-vocabulary word model and flexible parsing. N o n . V e r b a l S o u n d M o d e l s Models for sounds other than speech have been shown to significantly increase performance of HMM-based recognizers for noisy input. [2] [4] In this technique, additional models are added to the system that represent non-verbal sounds, just as word models represent verbal sounds. These models are trained exactly as ff they were word models, but using the noisy input. Thus, sounds that are not words are allowed to map onto tokens that are also not words. O u t o f v o c a b u l a r y W o r d M o d e l This module has not yet been implemented, In order to deal with out-of-vocabulary words, we will use a technique essentially like the one presented by BBN. [5] We will create an explicit model for out-of-vocabulary words. This model allows any triphone (context dependent phone) to follow any other triphone (given of course that the context is the same) with a bigram

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Expanding the Scope of the ATIS Task: The ATIS-3 Corpus

The Air Travel Information System (ATIS) domain serves as the common evaluation task for ARPA"spoken language system developers. 1 To support this task, the Multi-Site ATIS Data COllection Working group (MADCOW) coordinates data collection activities. This paper describes recent MADCOW activities. In particular, this paper describes the migration of the ATIS task to a richer relational database...

متن کامل

CMLPs robust spoken language understanding system

This paper outlines the general strategies followed in developing the CMU (Carnegie Mellon University) speech understanding system. Our system is oriented toward the extraction of information relevant to a task. It uses a exible frame-based parser. Our system handles phenomena that are natural in spontaneous speech, for example, restarts, repeats and grammatically ill-formed utterances. It main...

متن کامل

Session 3: Spoken Language Systems III

Since February 91, the ATIS database has been updated and, in an effort to quickly collect a larger amount of training and test data, a concerted effort has taken place in collecting data at five different sites (AT&T, BBN, CMU, MIT, and SRI). (See [1] for details.) About 10,000 spontaneous utterances were collected, of which about half were annotated (text transcriptions, reference answers, et...

متن کامل

DARPA ATIS Test Results June 1990

Introduction The first Spoken Language System tests to be conducted in the DARPA Air Travel Information System (ATIS) domain took place during the period June 15 20, 1989. This paper presents a brief description of the test protocol, comparator software used for scoring results at NIST, test material selection process, and preliminary tabulation of the scored results for seven SLS systems from ...

متن کامل

Session 5: Overview of the ATIS System

Two Sessions were devoted to introducing the DARPA SLS Common Task: the Air Travel Information System (ATIS). The first of these (in the late afternoon) provided an overview of the ATIS task and presented a summary of results reported to NIST in the first of a projected series of ATIS benchmark tests, while the second (immediately after dinner) served to describe different approaches in impleme...

متن کامل

SRI's Experience with the ATIS Evaluation

SRI International participated in the June 1990 Air Travel Information System (ATIS) natural-language evaluation. This report briefly describes the system that SRI used in the evaluation, analyzes SRI's results, and makes some recommendations for changes in the database structure and data collection system to be used for future ATIS evaluations.

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 1991